1. Field of the Invention
The invention relates generally to reducing the size of stored data and more specifically to reducing the size of information used to manage a data processing system.
2. Description of Related Art: FIGS. 1 and 2
Nothing is easier in a modem data processing system than storing data, and nothing is cheaper than the storage media needed to store it. Yet one of the chief consequences of this happy situation is that system managers are forever short of storage for their data. No one likes to actually throw data out, so many techniques have been developed to reduce the size of the data. Size reduction techniques fall into two classes: lossless techniques, in which the size of the data is reduced but no information is lost, and lossy techniques, in which the size of the data is reduced and some of the information is lost, but the interesting information is saved. Both techniques have their advantages and disadvantages. Lossless techniques save all the information but reduce its size by encoding the information. The need to preserve all the information limits the amount of reduction that can be achieved and the need first to encode the information, then to decode the information whenever it is read and then to decode and encode it whenever it is altered greatly increases the overhead of working with such information. Far greater size reductions are possible with lossy techniques than with lossless techniques and encoding can often be avoided, but lossy techniques are based on saving interesting information and therefore tend to be application-dependent, since it is the application that defines what is interesting.
An example of lossy size reduction techniques is the roll up tables that are used in database management systems to reduce the amount of system management information that must be stored. By reducing the amount of information that must be stored, roll up tables also make management of the information easier. For example, queries execute much faster on the roll up tables than they would have on the equivalent non-rolled up tables. FIG. 1 is an overview of a modem DBMS system management system 101, the Oracle® Enterprise Manager, manufactured by Oracle Corporation, Redwood City, Calif. System management system 101 includes a management service 113 that resides on a management server 111, a computer system that includes an Oracle DBMS and network connections to a group of monitored targets 103(i..n). Typically, monitored targets 103 are Oracle DBMS systems and Oracle servers which receive requests for information contained in the Oracle DBMSs from users of the World Wide Web.
Each monitored target 103(i) includes a management agent 105 which is an agent for management service 113. Agent 105 continually monitors its target 103(i) as specified by commands that agent 105 receives from management service 113 and sends messages about the results of its monitoring to management service 113. The occurrences in target 103(i) that agent 105 monitors are termed in the following events. An event may be a hit on a Web page that is stored on target 013(i), it may be a system condition that has crossed a threshold, such as the amount of disk space available, or any other occurrence that is of interest to management service 113. Commands from service 113 and messages from agent 105 are communicated using Internet protocols across a network connecting management service 113 and target 103(i). Management service 113 may respond to a message by storing its contents in tables in management repository 115, an Oracle DBMS. Management service 113 may of course also take immediate action in response to a message. The immediate action may be an automatic response to the message or it may be providing a message to central console 121, which a database administrator (DBA) uses to communicate with management service 113. The DBA may then use console 121 to enter commands to management service 113 to deal with the situation noted by management service 113. The DBA may also use console 121 to investigate the current and historical state of the monitored targets 103 and to reconfigure the monitored targets.
If there is any large number of monitored targets 103, the agents return an enormous amount of information to management service 113 and most of the returned information ends up in management repository 115. Consequently, management repository 115 quickly fills up. To reduce the amount of space required by the returned information in repository 115, management service 113 periodically aggregates the older returned information to reduce its size. To aggregate the information, management service 103 rolls up the older returned information to produce rolled up information which is much smaller than the information it was made from and then replaces the older information with the rolled up information. Thus, in FIG. 1, management repository 115 includes current non-rolled up information 117 and less current rolled up information 119.
Since the rolled up information is on the one hand historical but on the other hand needs to be easily accessible to central console 121, management service 103 uses lossy techniques to do the roll up. FIG. 2 gives a simplified example. The events being monitored in the example are hits on Web pages. For purposes of the example, management repository 215 is taken to include a page hit table 201 which has entries that record accesses by users on the World Wide Web to Web pages provided by monitored targets 103. There is a hit entry 209 for each hit on a page in a monitored target 103. Each entry includes three items of data: the URL (universal resource locator) for the page, a time/date stamp 205 which indicates when the hit occurred, and the source Internet address 207 of the entity that made the hit.
Clearly, page hit table 201 will grow very rapidly. Management service consequently periodically rolls up table 201 to produce a page hit roll up table 211 for a period X. Roll up table 211 contains only two columns: one, 213, for page URLs and one, 215, that indicates the number of hits received on the page during the period X. There is only one entry for each of the page URLs in table 211, and the value of field 215 for the entry is the number of hits experienced by the page in the period X. As will be immediately apparent from the foregoing, management service 113 makes table 211 from table 201 by making a single entry 217 in table 211 for each of the URLs that is present in table 201, counting the number of entries for each URL in table 201 for the period X, and placing the result of the count in no. of hits field 215. As will also be immediately apparent, table 211 is far, far smaller than table 201. In the following, tables like page hit roll up table 211 will be called aggregation tables and their entries aggregated entries, since each entry in table 211 may aggregate information from many entries in table 201. Further, values such as number of hits 215 which are made by combining a set of values such that the individual values in the set are lost will be termed herein metric values. Other examples of such metric values are averages, maxima, minima, modes, and medians. The meaning of a metric value of course depends on the kind of event. For example, if the event indicates that a condition to which the DBA must respond has arisen, the metric value may indicate the time between the time at which the event occurred and the time at which the DBA responded, and the aggregated metric value may be the average response time.
Of course, table 211 may itself be rolled up. For example, if the period X is one day and there is thus a roll up table 211 for each day, a weekly roll up table may be made from seven daily tables 111. Again, there would be one entry for each URL upon which there was a hit during the week, and no. of hits 215 would contain the number of hits for the week. The week tables may be rolled up into month tables, the month tables into year tables, and so on. The creation of any roll up entry may be regarded as a roll up event at a roll up level n, with the entry created by the roll up being a roll up event entry for level n and the roll up at level n+1 rolling up the roll up event entries for level n.
Aggregation tables are challenging to design. The challenge is to reduce the size of the information in the aggregation table as much as possible while reducing the usefulness of the information contained in the table as little as possible. Table 211 illustrates the difficulty. In page hit table 201, the time at which each hit occurred is recorded in time/date field 205; this information is lost in table 211; thus, though table 211 can tell the DBA how many times a page was hit in the period X, it cannot tell the DBA anything about the temporal distribution of hits over the period X. This pattern information may, however, be exactly what the DBA needs to correctly distribute copies of the page among monitored targets 103. What is needed if rollup table 211 is to provide useful information about the temporal distribution of the hits is a way of representing time/date information 105 in aggregated page hit entry 217 for the individual hit entries 209 that have been aggregated into entry 217. In more general terms, the problem is this: how to incorporate information that consists of sets of values into aggregation tables. What is needed, and what is provided by the invention disclosed herein, is a technique for doing this.